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One Virtual Machine 


Nearly Unlimited QA environments 
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Our development workflow 


e Two week sprints 
e New code is committed to a branch in git 


e When coding is complete, a Pull Request is opened in GitHub for review 


Branch can be staged onto a beta server for QA 


PR is merged into a branch named develop when code review and QA is 
complete 


The develop branch is merged into the master branch when it's time for 


a deployment 
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Discontinuous Integration (the old way) 


ssh betaserver 

cd /www/beta5 

source bin/activate 

git fetch origin && git reset --hard && git checkout -t origin/my-branch 
pip install --upgrade -r requirements.txt 

(cd frontend && npm install && gulp build) 

importdb atl db beta5 

django-admin collectstatic -L --noinput 

touch wsgi.py 

ohpleasegod --help 
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Discontinuous Integration 
Why was this bad? 


e Tedious! 
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Discontinuous Integration 
Why was this bad? 


e Tedious! 
e Error prone! 


e Stateful! 


© © © old-timey-beta-setup (ssh) 


user@betaserver /www/beta5/live git status 
On branch my-awesome-feature-branch 
Changes not staged for commit: 
(use "git add <file>..." to update what will be committed) 


(use "git checkout -- <file>..." to discard changes in working directory) 
modified: frontend/config.js 
modified: requirements.txt 


Untracked files: 
(use "git add <file>..." to include in what will be committed) 


apps/somebody elses old code/ 


no changes added to commit (use "git add" and/or "git commit -a") 


user@betaserver /www/beta5/live 
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Discontinuous Integration 
Why was this bad? 


e Tedious! 
e Error prone! 
e Stateful! 


e Enough instances to waste resources, not enough to ensure 
one is always available 
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Discontinuous Integration 
Why was this bad? 


e Tedious! 
e Error prone! 
e Stateful! 


e Enough instances to waste resources, not enough to ensure 
one is always available 


e Worst of all, NGINX played only a minor role 
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We can do better! 
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Business Need 


e Developers need to easily stage their work for peer and 
stakeholder review. 
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Beta architecture overview 
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Infinite" beta environments—components 


e Jenkins 

e Reflinks 

e supervisord managing uWSGI emperor mode 
e nginx-mod-passenger for NodeJS apps 

e NGINX and uWSGI socket files 

e Database snapshots 


e External proxy 
* ish—as many as we need "Atlantic 


New process 


Two common use-cases, with different requirements: 
1. Simple bug fixes and minor features 


2. Long-running feature branches 
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New process—typical setup 


O (1 Opena pull reguest 


Jenkins build (roughly 10 minutes) 


EL 


https://pr-1234.beta.theatlantic.com ü [J] 
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Pipeline 


theatlantic < 10 


OK 
0b69a18 og 


Pull Request: PR-... Z 


Commit: 


Start checkout build 


/ O 


citylab_sponsore 


frontend 


legacy. pipeline 


platform 


python 


Changes Tests Artifacts 


Changes by Obssa Bizuwork 
Replayed #9 


collectstatic docs test 


frontend 


selenium 


© EE ke 


post-test 


coverage report 


rsync develop 


stage beta 


X 


End 


VA theatlantic < 10 Pipeline Changes Tests Artifacts >] X 


Pull Request: PR-... Z  Q 10m 15s Changes by Obssa Bizuwork 


Commit: 0b69a18 © afew seconds ago Replayed #9 


Start checkout build collectstatic docs test post-test End 


v 


Y v / / 


citylab_sponsored coverage report 


/ 


frontend django rsync develop 


v 


legacy pipeline frontend stage beta 


A Y 


platform selenium 


python 


New process— named betas 


e Uses a distinct, persistent copy of the database (to allow for 
schema changes and feature-specific data) 


e Accessible outside the internal network with authentication 
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/ 


New process— named betas 


Q p Push out a branch: beta/my-feature 


Jenkins build (roughly 10 minutes) 


E] 


https://my-feature.beta.theatlantic.com chy D! zi 
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Business Value 


e Efficiency: Cuts down on developer time spent on tasks 
unrelated to business goals. 


« Morale: Automates repetitive/monotonous work. 


e Improves Coverage/Consistency: No barriers to manual 
testing, even for trivial changes. Each build environment is 
created with the exact same process. 
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Infinite" beta environments—components 


e Jenkins 


Reflinks 


supervisord managing uWSGI emperor mode 
e nginx-mod-passenger for NodeJS apps 
e NGINX and uWSGI socket files 


e Database snapshots 


External proxy 
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Jenkins—stage beta 


SY theatlantic < 10 Pipeline Changes Tests Artifacts e 5 oa 3] X 


Pull Reguest: PR-... Z © 10m15s Changes by Obssa Bizuwork 


Commit: Ob69a18 © a few seconds ago Replayed #9 


Start checkout build collectstatic docs test post-test End 


/ 


Y / 


v 


citylab_sponsored coverage report 


v / 


frontend django rsync develop 


/ 
legacy_pipeline frontend 
/ / 
platform selenium 


python 


Atlantic 


Jenkinsfile 


e A declarative build and deployment script, committed to 
source control in the root of the repo 


e Hooked into github notifications for pull request opens and 
branch pushes 


e Reasonably powerful control over conditional logic, 
parallelization, pass/fail conditions, etc. 
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Jenkinsfile 


pipeline 4 


// 


stage('stage beta!) { 


when { anyOf { 
expression { BRANCH NAME === /PR-[0-9]+/ | // a pull request 
branch 'develop' // development branch 
expression { BRANCH, NAME.startsWith('beta/') | // 'beta/feature' 

P 


steps 1 
// Initialize database 
sh """ ssh devdbserver "create beta db SBRANCH NAME" MIN 
// If the workspace doesn't exist, create a reflink copy 
sh """ssh betaserver \\ 


"[ -d S(WORKSPACE) | || cp -ar --reflink=auto AN 
/www/builds/theatlantic/develop.base SÍWORKSPACE;" "un 
// ...continued 
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Jenkinsfile (continued) 


IT 2-3 
stage('stage beta!) 4 
If ras 
steps (1 
// Copy current build to remote workspace dir 
sh """rsync -acvzhO --delete AN 
--exclude "x.pyc" --exclude .git --exclude ... \\ 
SWORKSPACE/ betaserver:SWORKSPACE/ "'"" 
// Create wsgi files, which will initialize uWSGI emperor processes 
sh """ssh betaserver "SWORKSPACE/support/beta/create wsgi.py" """ 
j 
j 
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Infinite" beta environments—components 


e Jenkins 

e Reflinks 

e supervisord managing uWSGI emperor mode 
e nginx-mod-passenger for NodeJS apps 

e NGINX and uWSGI socket files 

e Database snapshots 


e External proxy 
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Jenkins—stage beta 


e |nitializes with a reflink copy of a previous build to save on 
disk space (1.2G => 3M) 


e Uses rsync to copy the newly created environment to the 
Detas server over top of this copy 
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Reflinks (copy-on-write) 


e BIRFS on debian or ubuntu 
e XFS on RHEL, CentOS, or Fedora 
e mkfs.xfs must be called with reflink=1 
e cp -ar --reflink-auto 
e only uses the space required for differences between builds 


e considered "experimental"—requires custom-built kernel 
(may be enabled by default in Fedora 29) 
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Infinite" beta environments—components 


e Jenkins 

e Reflinks 

e supervisord managing uWSGI emperor mode 
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External proxy 
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uWSCl emperor + supervisord 


[program:zerg server | 
command = /sbin/uwsgi 
--master --thunder- Lock 
--emperor "/www/builds/theatlantic/*/uwsgi/master.*.1in1" 


autorestart - true 


[program:workers | 
command = /sbin/uwsgi 
--master --thunder- Lock 
--emperor "/www/builds/theatlantic/*/uwsgi/workers.*.17n1'" 


autorestart = true 
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Infinite" beta environments—components 


[e] A — 


uWSGI zerg master uWSGI zerg workers 
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uWSCl emperor + supervisord 


; /www/builds/theatlantic/pr-1234/uwsgi/master.theatlantic.ini 

[uwsgi | 

; Yn: current config filename, without the .ini extension 

; Strip off the leading 'master.' part of the filename to determine the app. 
app = @(exec:///usr/bin/perl -e '$_ = "?n"; s//master\.//g; print') 


subdomain = %3 ; the 4th path component (0-indexed); eg pr-1234 
thunder-lock - true 
master - true 


processes - 0 


http = /var/run/uwsgi/%(app)-%(subdomain).sock 
zerg-server = /var/run/uwsgi/%(app)-%(subdomain)-workers.sock 


Atlantic 


uWSGI zerg server 


; /www/builds/theatlantic/pr-1234/uwsgi/master.theatlantic.ini 

[uwsgi | 

; Yn: current config filename, without the .ini extension 

; Strip off the leading 'master.' part of the filename to determine the app. 
app = @(exec:///usr/bin/perl -e '$_ = "%n"; s/^masterN.//g; print') 


subdomain = %3 ; the 4th path component (0-indexed); eg pr-1234 
thunder-lock - true 


master - true 
processes - 0 


http = /var/run/uwsgi /?$( app) -% (subdomain) .sock 
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uWSGI zerg workers 


; /www/builds/theatlantic/pr-1234/uwsgi/worker.theatlantic.ini 
[uwsgi | 

; attach to zerg pool 

zerg = /var/run/uwsgi/%(app)-%(subdomain)-workers.sock 


cheaper-algo - busyness 


cheaper - 1 ; minimum number of workers to keep at all times 
cheaper-initial - 2 ; starts with minimal workers 
cheaper-step - 2 ; spawn at most 2 workers at a time 


cheaper-overload = 30 ; seconds between busyness checks 
cheaper-busyness-multiplier - 50 
cheaper-busyness-penalty - 2 


; how many requests are in backlog before quick response triggered 
cheaper-busyness-backlog-alert - 33 

cheaper-busyness-backlog-step - 1 

idle = 86400 ; workers shut down if the beta is idle for a day 


wsgi-file = %(env_dir)/apps/wsgi.py 
env = DJANGO SETTINGS MODULE=settings.%(app).live 
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NGINX, proxying to UWSGI socket files 


server 4 
server name "~4(?<subdomain>.+)\.beta\.theatlantic\.coms"; 


root /www/builds/theatlantic/$subdomain; 


location / { 
try files /assets/Suri @django; 
j 


location @django { 
internal; 
include includes/proxypass.conf; 
proxy pass http://unix:/var/run/uwsgi/theatlantic-$subdomain.sock; 
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nginx-mod-passenger for nodejs 


server 1 
server name "~4(?<subdomain>.+)\.ampbeta\.theatlantic\.coms"; 


root /www/builds/amp/Sisubdomain) /public; 


passenger enabled on; 

passenger app type node; 

passenger app root /www/builds/amp/$(subdomain); 
passenger restart dir /www/builds/amp/${subdomain} ; 
passenger startup. file dist/host/bin/www.js; 
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Guestions? 
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Thank you! 


github.com/theatlantic/nginxconf-2018 


frankie@theatlantic.com 
mhowsden@theatlantic.com 
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